fix(metabench): isolate native probes and harden benchmark reporting by geeknoid · Pull Request #807 · microsoft/oxidizer

Martin Taillefer (geeknoid) · 2026-10-11T18:42:50Z

Summary

Move allocation, perf, and VTune collection into private native-case callers, preserving benchmark bodies and collector-free ordinary adapters.
Bypass allocation tracking in ordinary workers and isolate each collector by validated worker mode.
Correct Criterion estimate/confidence reporting, native case identities, compiler compatibility, and baseline publication safety.
Harden artifact admission and diagnostic output; add structural, protocol, lifecycle, cross-thread, and failure-path regressions.
Require Linux native listing/smoke contracts in CI and document the supported measurement boundaries.

Validation

Scoped Anvil Clippy and spelling checks, changed-file formatting, and generated README verification passed.
123 runtime unit tests, public API/protocol tests, macro structural and consumer contracts, and targeted native process regressions passed.
Required Linux listing/smoke checks passed; missing native prerequisites fail the required tier.
Miri, mutation testing, full-workspace tests, and performance measurements were not run locally.

Copilot

🟡 Changes recommended

Captured worker logs remain unbounded and can exhaust filesystem storage.

1 open finding

Unbounded worker logs can exhaust disk space · New

What changed in this PR

This PR isolates Metabench collectors in native probe workers and strengthens reporting, artifact handling, diagnostics, and CI coverage.

Changes:

Adds generated native probe discovery/execution with isolated allocation, perf, and VTune collection.
Hardens report metadata, Criterion estimates, artifact admission, baseline publication, and diagnostics.
Adds extensive protocol, lifecycle, failure-path, and Linux native-contract tests.

File	Description
`.github/workflows/repository-checks.yml`	Adds required Linux native-contract checks.
`Cargo.toml`	Updates shared Gungraun-related dependencies.
`Cargo.lock`	Records dependency resolution changes.
`docs/metabench-native-contracts.md`	Documents native CI contracts.
`crates/metabench/Cargo.toml`	Updates Metabench dependencies.
`crates/metabench/README.md`	Regenerates crate documentation.
`crates/metabench/build.rs`	Captures build-time compiler identity.
`crates/metabench/src/__private.rs`	Exposes generated probe support.
`crates/metabench/src/allocation.rs`	Activates tracking only in allocation workers.
`crates/metabench/src/allocator.rs`	Adds collector-aware allocator wrapper.
`crates/metabench/src/arguments.rs`	Adds smoke-mode argument handling.
`crates/metabench/src/artifact.rs`	Hardens artifacts and estimate parsing.
`crates/metabench/src/entrypoint.rs`	Connects generated probes to the runner.
`crates/metabench/src/error.rs`	Adds probe-specific diagnostics.
`crates/metabench/src/identity.rs`	Removes obsolete allocation-selection accessor.
`crates/metabench/src/lib.rs`	Documents measurement and reporting contracts.
`crates/metabench/src/metric_display.rs`	Updates metric labels and ordering.
`crates/metabench/src/perf.rs`	Activates perf only in validated workers.
`crates/metabench/src/probe.rs`	Implements probe discovery and execution.
`crates/metabench/src/report.rs`	Adds schema-9 estimates and safer rendering.
`crates/metabench/src/runner.rs`	Orchestrates probes, logs, smoke runs, and publication.
`crates/metabench/src/vtune.rs`	Activates VTune only in validated workers.
`crates/metabench/examples/instrumentation_controls.rs`	Adds instrumentation comparison fixture.
`crates/metabench/examples/measurement_boundaries.rs`	Adds workload-boundary fixture.
`crates/metabench/examples/native_probe_contracts.rs`	Adds lifecycle and iterator fixture.
`crates/metabench/examples/threaded_allocations.rs`	Adds cross-thread allocation fixture.
`crates/metabench/tests/probe_protocol.rs`	Tests authenticated probe rejection paths.
`crates/metabench/tests/spawned_benchmark.rs`	Expands process-level regressions.
`crates/metabench/tests/fixtures/artifacts/criterion/duplicate/a/new/estimates.json`	Updates Criterion estimate fixture.
`crates/metabench/tests/fixtures/artifacts/criterion/duplicate/b/new/estimates.json`	Updates Criterion estimate fixture.
`crates/metabench/tests/fixtures/artifacts/criterion/invalid_numeric/new/estimates.json`	Updates invalid estimate fixture.
`crates/metabench/tests/fixtures/artifacts/criterion/minimal/new/estimates.json`	Updates minimal estimate fixture.
`crates/metabench/tests/fixtures/artifacts/criterion/missing_function/new/estimates.json`	Updates missing-function fixture.
`crates/metabench/tests/fixtures/artifacts/criterion/unicode/new/estimates.json`	Updates Unicode fixture.
`crates/metabench/tests/fixtures/artifacts/traversal/cycle/new/estimates.json`	Updates traversal fixture.
`crates/metabench_macros/src/lib.rs`	Documents collector-free macro expansion.
`crates/metabench_macros/tests/compile.rs`	Reworks compile-test fixture reuse.
`crates/metabench_macros/tests/ui/pass/generated_name_collisions.rs`	Expands generated-name collision coverage.
`crates/metabench_macros_impl/src/benchmark.rs`	Generates isolated probe adapters.
`crates/metabench_macros_impl/src/shared.rs`	Adds readable collision-safe native names.
`crates/metabench_macros_impl/src/target.rs`	Generates probe tables and lifecycle links.

🧠 Review effort: Balanced

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Martin Taillefer (geeknoid) · 2026-10-11T18:47:53Z

Fixed the Windows CI baseline failures: artifact-admission tests now respect the reader’s intentional write-sharing denial, and the oversized fixture closes its writer before admission. The actual stream-size bound remains independently tested. Amended and pushed the commit; Windows CI will verify the platform-specific path.

Copilot

🟡 Changes recommended

Markdown and baseline diagnostics still permit terminal-direction manipulation through unsanitized text.

3 open findings

🧠 Review effort: Balanced

Copilot

🟡 Changes recommended

Schema-9 validation accepts confidence bounds without their required confidence level.

1 open finding

Reject confidence intervals with null confidence levels · New

3 resolved since last review

🧠 Review effort: Balanced

Copilot

🟡 Changes recommended

Repository report generators remain incompatible with schema 9, and the documented native smoke behavior differs from the implementation.

3 open findings

🧠 Review effort: Balanced

Copilot

🔵 Needs a closer look

It combines proc-macro generation, unsafe allocator routing, cross-platform process control, and report-schema migration without full workspace, Miri, or mutation validation.

0 open findings

3 resolved since last review

🧠 Review effort: Balanced

Copilot

🟡 Changes recommended

Schema-9 validation still permits Criterion timing metrics without the statistic needed for safe baseline comparison.

1 open finding

Require statistic for criterion/time metrics in schema 9 · New

🧠 Review effort: Balanced

Martin Taillefer (geeknoid) · 2026-10-12T05:17:55Z

Both Linux mutation runs now report zero surviving mutations (306 caught, 51 unviable). Their only remaining failure was a timeout when the worker deadline was disabled. Published a476dd6 adds a finite fake-worker regression and a two-second test-child backstop, so that defect fails an assertion instead of hanging the suite. Production behavior and benchmark bodies are unchanged; no additional mutations are excluded. Fresh CI will verify the result.

Copilot

🔵 Needs a closer look

It substantially changes macros, process orchestration, artifact security, and platform-specific collectors without full workspace, Miri, or mutation validation.

0 open findings

🧠 Review effort: Balanced

Sander Saares (sandersaares) · 2026-10-12T08:33:01Z

Pull request overview (auto-generated).

Martin Taillefer (geeknoid) · 2026-10-12T09:51:47Z

Addressed the Windows mutation result (3 survivors and 1 timeout) in 06496c4:

Assert private run-directory creation directly on every host, rather than relying only on Unix permission checks.
Test reparse-point rejection with actual attribute values, without requiring Windows symlink privileges.
Exercise the perf failed-start bookkeeping in an isolated subprocess on Windows as well as Linux.
Give the Windows empty-pipe regression its own subprocess deadline, so a blocking-read mutation fails promptly rather than hanging the entire suite.

Benchmark workloads and collector behavior are unchanged; no mutation exclusions were added. Linux unit/lint checks passed. Local Windows cross-compilation was blocked by the dependency C build requiring MSVC tools, so Windows validation remains with CI. The new head also retests the previously failed, unchanged ARM allocator test.

Copilot

🔵 Needs a closer look

The broad cross-platform process, allocator, proc-macro, artifact, and CI changes warrant final human review despite extensive regression coverage.

0 open findings

🧠 Review effort: Balanced

Sander Saares (sandersaares)

[Copilot speaking]

Published 15 findings. No finding follows up on an existing discussion thread.

See diagnostics

Diagnostic	Value
Cache	Miss

martin-kolinek

🤖: Approved.

Copilot

🔵 Needs a closer look

The cross-platform process, allocator, and macro changes warrant human review alongside the unrun full-workspace, Miri, and mutation checks.

0 open findings

🧠 Review effort: Balanced

- Keep benchmark bodies and ordinary adapters free of collection controls by generating private allocation, perf, and VTune callers from native cases. - Gate alloc_tracker behind the same underlying allocator instance and bypass tracking in ordinary workers; isolate collectors by validated worker mode. - Report Criterion slope or mean estimates with confidence metadata, validate bootstrap intervals, and compare baselines using the build-time compiler. - Preserve explicit native case identities, use executable-derived artifact roots, and prevent empty, smoke, failed, or partial runs replacing baselines. - Bound artifact reads using regular-file handles, create worker logs exclusively, and encode less-trusted terminal and Markdown output. - Add structural macro, indexed protocol, lifecycle, cross-thread allocation, and collector-failure regressions, including generated-name collision checks. - Require Linux native listing and smoke contracts in CI and update generated crate documentation and native-contract guidance. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2cbef463-84aa-4249-91e4-20ba418301f6

Martin Taillefer (geeknoid) · 2026-10-12T15:10:28Z

Rebased the existing commit onto current main (131f7d6), with owner approval, and published ea6c341. The preceding head had no PR CI runs because four merge conflicts prevented GitHub from creating the test merge commit; the green dynamic run was only Copilot review, not CI.

Combined all upstream mutation test selections with the metabench-specific stress-test skip, and preserved main’s removal of the retired infinity-pool comparison rows while retaining the primary Criterion time metric. Metabench implementation files and all public/macro-support signatures are byte-for-byte unchanged by the rebase. The conventional commit body and trailers are preserved. Awaiting CI on this conflict-free head.

Copilot

🔵 Needs a closer look

The extensive cross-platform process, allocator, artifact-security, and proc-macro changes warrant final human review.

0 open findings

🧠 Review effort: Balanced

martin-kolinek

🤖: Approved.

Copilot AI balanced review requested due to automatic review settings October 11, 2026 18:42

Copilot started reviewing on behalf of Martin Taillefer (geeknoid) October 11, 2026 18:43

View session

Copilot AI reviewed Oct 11, 2026

View reviewed changes

Comment thread crates/metabench/src/runner.rs Outdated

Copilot AI balanced review requested due to automatic review settings October 11, 2026 18:47

Martin Taillefer (geeknoid) force-pushed the fix/metabench-native-probes branch from 5ae2b8f to 6a43159 Compare October 11, 2026 18:47

Copilot started reviewing on behalf of Martin Taillefer (geeknoid) October 11, 2026 18:48

View session

Copilot AI reviewed Oct 11, 2026

View reviewed changes

Comment thread crates/metabench/src/report.rs Outdated

Comment thread crates/metabench/src/runner.rs Outdated

github-advanced-security Bot found potential problems Oct 11, 2026

View reviewed changes

Comment thread crates/metabench/src/allocator.rs Fixed

Comment thread crates/metabench/src/allocator.rs Fixed

martinhavelka (wukchung) reviewed Oct 11, 2026

View reviewed changes

Comment thread crates/metabench/src/runner.rs

Comment thread crates/metabench/src/runner.rs Outdated

Copilot AI balanced review requested due to automatic review settings October 11, 2026 19:16

Martin Taillefer (geeknoid) force-pushed the fix/metabench-native-probes branch from 6a43159 to 187b972 Compare October 11, 2026 19:16

Copilot started reviewing on behalf of Martin Taillefer (geeknoid) October 11, 2026 19:16

View session

Martin Taillefer (geeknoid) enabled auto-merge (squash) October 11, 2026 19:18

Copilot AI reviewed Oct 11, 2026

View reviewed changes

Comment thread crates/metabench/src/report.rs Outdated

Copilot AI balanced review requested due to automatic review settings October 11, 2026 19:24

Martin Taillefer (geeknoid) force-pushed the fix/metabench-native-probes branch from 187b972 to 3d9af32 Compare October 11, 2026 19:24

Copilot started reviewing on behalf of Martin Taillefer (geeknoid) October 11, 2026 19:24

View session

Copilot AI reviewed Oct 11, 2026

View reviewed changes

Comment thread crates/metabench/src/report.rs

Comment thread crates/metabench/src/lib.rs Outdated

Martin Taillefer (geeknoid) force-pushed the fix/metabench-native-probes branch from 3d9af32 to 28320ec Compare October 11, 2026 19:35

Copilot AI balanced review requested due to automatic review settings October 11, 2026 19:35

Copilot started reviewing on behalf of Martin Taillefer (geeknoid) October 11, 2026 19:35

View session

Copilot AI reviewed Oct 11, 2026

View reviewed changes

Copilot AI balanced review requested due to automatic review settings October 11, 2026 19:44

Martin Taillefer (geeknoid) force-pushed the fix/metabench-native-probes branch from 28320ec to cb5b75d Compare October 11, 2026 19:44

Copilot started reviewing on behalf of Martin Taillefer (geeknoid) October 11, 2026 19:44

View session

Copilot AI reviewed Oct 11, 2026

View reviewed changes

Comment thread crates/metabench/src/report.rs

Martin Taillefer (geeknoid) force-pushed the fix/metabench-native-probes branch from cb5b75d to 7bfd73f Compare October 11, 2026 19:55

Copilot AI balanced review requested due to automatic review settings October 11, 2026 19:55

Copilot started reviewing on behalf of Martin Taillefer (geeknoid) October 11, 2026 19:56

View session

Copilot started reviewing on behalf of Martin Taillefer (geeknoid) October 12, 2026 05:18

View session

Copilot AI reviewed Oct 12, 2026

View reviewed changes

Sander Saares (sandersaares) self-requested a review October 12, 2026 08:20

Copilot AI balanced review requested due to automatic review settings October 12, 2026 09:51

Martin Taillefer (geeknoid) force-pushed the fix/metabench-native-probes branch from a476dd6 to 06496c4 Compare October 12, 2026 09:51

Copilot started reviewing on behalf of Martin Taillefer (geeknoid) October 12, 2026 09:52

View session

Copilot AI reviewed Oct 12, 2026

View reviewed changes

Sander Saares (sandersaares) approved these changes Oct 12, 2026

View reviewed changes

martin-kolinek reviewed Oct 12, 2026

View reviewed changes

Comment thread crates/metabench/src/artifact.rs Outdated

martin-kolinek reviewed Oct 12, 2026

View reviewed changes

Comment thread crates/metabench/src/report.rs

martin-kolinek reviewed Oct 12, 2026

View reviewed changes

Comment thread crates/metabench/src/error.rs

martin-kolinek approved these changes Oct 12, 2026

View reviewed changes

Copilot AI balanced review requested due to automatic review settings October 12, 2026 13:15

Martin Taillefer (geeknoid) force-pushed the fix/metabench-native-probes branch from 06496c4 to 5b2a76b Compare October 12, 2026 13:15

Copilot started reviewing on behalf of Martin Taillefer (geeknoid) October 12, 2026 13:16

View session

Copilot AI reviewed Oct 12, 2026

View reviewed changes

Copilot AI balanced review requested due to automatic review settings October 12, 2026 15:10

Martin Taillefer (geeknoid) force-pushed the fix/metabench-native-probes branch from 5b2a76b to ea6c341 Compare October 12, 2026 15:10

Copilot started reviewing on behalf of Martin Taillefer (geeknoid) October 12, 2026 15:11

View session

Copilot AI reviewed Oct 12, 2026

View reviewed changes

martin-kolinek approved these changes Oct 12, 2026

View reviewed changes

Martin Taillefer (geeknoid) merged commit 13d95b8 into main Oct 12, 2026
40 checks passed

Martin Taillefer (geeknoid) deleted the fix/metabench-native-probes branch October 12, 2026 20:08

Ralf Biedert (ralfbiedert) mentioned this pull request Oct 13, 2026

perf(performables): avoid allocation in ready blocking operations #810

Open

Conversation

Martin Taillefer (geeknoid) commented Oct 11, 2026

Summary

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Martin Taillefer (geeknoid) commented Oct 11, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Martin Taillefer (geeknoid) commented Oct 12, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Sander Saares (sandersaares) commented Oct 12, 2026

Uh oh!

Martin Taillefer (geeknoid) commented Oct 12, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Sander Saares (sandersaares) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

martin-kolinek left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Martin Taillefer (geeknoid) commented Oct 12, 2026